-
Notifications
You must be signed in to change notification settings - Fork 713
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Introduce the Logstash operator for ECK #6732
Conversation
* Initial Commit of ECK for Logstash Initial commit of a simple operator. The first operator will create: * A Service exposing the logstash metrics API, so it can be used for monitoring purposes, with the option to expose further services if necessary * Secrets holding logstash.yml * A StatefulSet deploying the logstash pods * Pods with default readiness probe The sample logstash yml file as located in config/samples/logstash/logstash_svc.yaml will create: ``` ✗ kubectl tree logstash logstash-sample NAMESPACE NAME READY REASON AGE default Logstash/logstash-sample - 7m3s default ├─Secret/logstash-sample-ls-config - 7m3s default ├─Service/logstash-sample-ls-api - 7m3s default │ └─EndpointSlice/logstash-sample-ls-api-4dl2x - 7m3s default ├─Service/logstash-sample-ls-beats - 7m3s default │ └─EndpointSlice/logstash-sample-ls-beats-rw8sp - 7m3s default ├─Service/logstash-sample-ls-default - 7m3s default │ └─EndpointSlice/logstash-sample-ls-default-6vmhp - 7m3s default └─StatefulSet/logstash-sample-ls - 7m3s default ├─ControllerRevision/logstash-sample-ls-86448964 - 7m3s default ├─Pod/logstash-sample-ls-0 True 7m3s default ├─Pod/logstash-sample-ls-1 True 7m3s default └─Pod/logstash-sample-ls-2 True 7m3s``` And shows status: ``` ✗ kubectl get logstash logstash-sample NAME AVAILABLE EXPECTED AGE VERSION logstash-sample 2 2 2m35s 8.6.1 ``` Co-authored-by: Kaise Cheng <[email protected]> Co-authored-by: kaisecheng <[email protected]> Co-authored-by: Peter Brachwitz <[email protected]> Co-authored-by: Michael Morello <[email protected]>
Following the prior art, this commit integrates stack monitoring for Logstash by adding metricsbeat and filebeat as sidecars to deliver metrics and log to monitoring es cluster There is a sample resource in config/samples/logstash/logstash_stackmonitor.yaml to deploy and test. Co-authored-by: Rob Bavey <[email protected]> Co-authored-by: Michael Morello <[email protected]>
This PR allows configuration of Logstash pipeline.yml in CRD. pipeline.yml is an array of map of string/interface{} defining multiple pipelines. `.` in between the key is treated as string. Pipelines can be set from either inline in Pipelines or secret referencing in PipelinesRef. e2e test: TestPipelineConfigRefLogstash, TestPipelineConfigLogstash Co-authored-by: Rob Bavey <[email protected]> Co-authored-by: Michael Morello <[email protected]>
This commit adds extra unit tests for the Logstash Controller, including tests for config, controller, pods and services
…pod restart (#6674) This commit adds the ability to reload logstash pipelines when the pipeline changes, without triggering a full restart of the pod, leveraging Logstash's ability to watch pipeline definitions, and reload automatically if a change is discovered. A logstash config directory typically includes logstash.yml, pipelines.yml, jvm.options and log4j2.properties required to run logstash - while logstash can store the contents of a pipeline definition in any location, the pipelines.yml definition file, which states where these definition files are must be in the same config directory as the other setup files. To enables us to have a mixture of copied and generated files in this config directory, an initContainer is used, with a small script to prepare the config directory. The script copies the contents of the /usr/share/logstash/config into the a shared config volume Additionally softlinks are created into this volume from the pipelines.yml and logstash.yml secrets created by the logstash operator. Additionally, we now do not include changes to pipelines in the configuration hash that triggers a pod reload, but instead write the config.reload.automatic: true setting in to logstash.yml Co-authored-by: Peter Brachwitz <[email protected]> Co-authored-by: Thibault Richard <[email protected]>
This commit adds information about Logstash to the set of ECK telemetry Co-authored-by: Thibault Richard <[email protected]>
Fix Logstash service e2e tests This test was failing, because the readiness probe was hitting the port defined by the service rather than the default port. This commit updates the e2e test to set the config to serve the logstash api from the same port that the service specifies. Co-authored-by: Michael Morello <[email protected]>
This commit allows configuration of `elasticsearchRefs` in CRD to associate Elasticsearch clusters to Logstash. The connectivity info/ credentials are passed via environment variable to Logstash container for referencing in Logstash plugins: logstash-output-elasticsearch, logstash-input-elasticsearch and logstash-filter-elasticsearch. Environment variables taking `$clusterName_ES_` as a prefix are available. Co-authored-by: Michael Morello <[email protected]> Co-authored-by: Rob Bavey <[email protected]> Co-authored-by: Peter Brachwitz <[email protected]>
buildkite test this -f p=gke -m t=TestSamples,t=TestVersionUpgradeOrdering,E2E_TAGS=logstash 🐸 🐞 ❔ |
* Fix the sample tests for logstash service and stack monitoring The stack monitoring tests were failing as the builder was not creating the monitoring elasticsearch references. This was added, as well as a minor refactoring to split the builder into adding metrics and logging monitoring separately. This commit also calculates the the monitoring assocation counts correctly, as previously the metrics association was double counted. This commit also fixes the logstash service sample end to end test - the pipeline id in the yaml definition did not match that in the test. Co-authored-by: Michael Morello <[email protected]>
buildkite test this -f p=gke -m t=TestSamples,t=TestVersionUpgradeOrdering,E2E_TAGS=logstash 🐸 |
buildkite test this -f p=gke -m t=TestSamples,t=TestVersionUpgradeOrdering,E2E_TAGS=logstash |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since all the commits in this PR have already been reviewed, and the new E2E tests seem to pass, I think we can merge it as is.
@robbavey please let me know if you think that's not the case and if anything else should be reviewed by the ECK team.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM as @barkbay said, this looks like the code we already reviewed. Let us know if there is something specific to look for. Otherwise I am really looking forward to merging this to main. 🎉
elasticsearchRefs: | ||
- clusterName: production | ||
name: elasticsearch-sample | ||
# secretName: external-cloud-es-ref |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove? I think this is not yet supported.
commonv1.ObjectSelector `json:",omitempty,inline"` | ||
// +kubebuilder:validation:Required | ||
// +kubebuilder:validation:MinLength=1 | ||
ClusterName string `json:"clusterName,omitempty"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we add a comment explaining ClusterName?
// TODO refactor identical to Kibana client | ||
func NewLogstashClient(logstash v1alpha1.Logstash, k *test.K8sClient) (*http.Client, error) { | ||
var caCerts []*x509.Certificate | ||
// TODO: Integrate with TLS on metrics API |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I assume we capture these TODOs in issues somewhere, correct?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes - we are currently tracking in ingest-dev
, but will move to public issues shortly
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🚀 🧪 🍀
Co-authored-by: Peter Brachwitz <[email protected]>
@pebrc @barkbay @thbkrkr Thank you for the reviews, getting @kaisecheng and I to this point - from the Logstash side, I am happy to see this merged into |
Also, are there any release notes/changelog entries that you would like me to add? |
This is handled separately and we will ask you at that time to contribute to it, thank you. |
This adjusts the number of volumes expected from Beats sidecars in the Logstash Stack Monitoring unit tests. Why? Because we don't test PRs with an automatic merge of the main branch (🐛🐞), we missed that the tests in #6732 had to be updated to take into account the changes made by #6703, which adds a new temp volume to the Beats sidecars.
This commit introduces a technical preview of the Logstash Operator for ECK
The Logstash operator introduces a Logstash CRD:
Look and feel:
Logstash with a simple pipeline, pointing to an elasticsearch instance:
Produces the following resources:
This operator provides support for:
logstash.yml
inconfig
orconfigRef
sections of the CRDelasticsearchRefs
, and environment variable substitution to introduce those elasticsearch references into logstash pipelines.pipelines.yml
inpipelines
orpipelinesRef
sections of the CRD with support for pipeline definition in volume mountsmonitoring.logs.elasticsearchRefs
andmonitoring.metrics.elasticsearchRefs
Logstash nodes are created as StatefulSets - we expect in later versions of the logstash operator to support persistence in Logstash nodes, including persistent queues and dead letter queues.
A work in progress PR includes documentation and recipes on how to use this logstash operator. There are also samples in this PR located under
config/samples/logstash/*
Relates: #1453